[orc] Optimize configuration creating in orc file format #4716

zhangyazhe · 2024-12-16T08:20:24Z

Purpose

This PR optimizes the creation performance of OrcFileFormat, with an overall performance improvement of at least 500 times. Performance tests can be found in the test files.
In actual tests, before optimization, creating an OrcFileFormat took about 6ms, with approximately 7 calls per query, and the scan time was about 40ms+. Under high QPS conditions (50+), the scan time could surge to several hundred milliseconds. After optimization, creating an OrcFileFormat is controlled within about 0.2ms, with approximately 7 calls per query, and the scan time is about 2ms or less.
Optimization principle: Each time Hadoop config is created and set, it scans system files.

Tests

paimon-format/src/test/java/org/apache/paimon/format/orc/OrcFileFormatTest.java

Note: This PR is split from #4497

paimon-format/src/main/java/org/apache/paimon/format/orc/OrcFileFormat.java

…ch to a regular cache.

JingsongLi · 2024-12-16T11:15:34Z

paimon-format/src/main/java/org/apache/paimon/format/orc/OrcFileFormat.java

    public OrcFileFormat(FormatContext formatContext) {
        super(IDENTIFIER);
        this.orcProperties = getOrcProperties(formatContext.options(), formatContext);
-        this.readerConf = new org.apache.hadoop.conf.Configuration();


I just use new Configuration(false), and tests passed. Do you think it is ok to just use this way?

[orc][format] add cache for OrcFileFormat

ae9cd45